Perceived-latency P2 + P3.5 servlet/HTTP2: hydrate, SWR cache, skeletons, Vary, h2c/h2 opt-in by harshach · Pull Request #28015 · open-metadata/OpenMetadata

harshach · 2026-05-10T16:24:15Z

Describe your changes:

Phase 2 of the perceived-latency work tracked in .context/perceived-latency-design.md, plus a small P3.5 servlet correctness fix and an opt-in HTTP/2 path at Jetty.

P2.3 — Batch lineage hydration endpoint (POST /v1/lineage/hydrate)
Server primitive: accepts a list of (type, id) pairs and returns hydrated entities grouped by entityType. Replaces N per-node entity GETs with one round-trip. Each entity is authorized individually with VIEW_BASIC; entities the caller cannot read are silently dropped from the response (rather than failing the whole batch). Useful for any client that needs to hydrate a graph of mixed entity types.

The original design hypothesized 50× per-node entity GETs in the Lineage view. On inspection the actual per-node fetches in the current Lineage code are for getTestCaseExecutionSummary (a different endpoint), not entity GETs, so this PR ships only the server-side primitive and the matching lineageAPI.ts helper.

P2.2 — Explore tab-switch SWR cache
Tab-switch on /explore re-runs the same shape of search-fetch with a different searchIndex. This adds a 30s stale-while-revalidate cache keyed by the same dependency string the page already uses to detect "should I refetch?". Cache hit renders synchronously with no spinner, then revalidates silently in the background. Capped at 60 entries.

P2.1 — Skeleton screens (TableDetail, MyData, Lineage)
Replace top-level <Loader /> gates with skeleton placeholders that mirror the page chrome. Cheap implementation: pure antd Skeleton primitives.

P3.5 — OpenMetadataAssetServlet correctness fixes
Two real bugs surfaced during the transport-layer audit:

Missing Vary: Accept-Encoding on compressed responses → CDN cache poisoning risk.
Broken q-value parsing — enc.contains("q=0") matched q=0.5 etc. Fixed by parsing as a double per RFC 7231 §5.3.4. Also tightened coding-name match (so brand no longer matches br) and added wildcard handling.

Three new unit tests cover the fixes; the existing 9 tests continue to pass.

P3.5 — Opt-in HTTP/2 at Jetty (type: h2 / type: h2c)
Adds dropwizard-http2 to openmetadata-service so type: h2 (TLS) and type: h2c (cleartext) connector factories are available. Operators opt in by uncommenting the example blocks in conf/openmetadata.yaml. Both connector types compose HTTP/1.1 + HTTP/2 protocol handlers on the same port (HTTP/1.1 clients still work via Upgrade / ALPN), so adopting them is backward-compatible.

Verified backward compatibility by temporarily switching the IT bootstrap connector to type: h2c and running:

SystemResourceIT → 34/34 pass
LineageHydrateIT → 5/5 pass

39/39 IT tests passed against the h2c connector — proves the OkHttp3 SDK works transparently via HTTP/1.1 Upgrade. The shipped IT YAML is unchanged (still type: http); only operators who explicitly opt in get Jetty-side HTTP/2.

Why HTTP/2 at Jetty matters even with an LB: end-to-end multiplexing on the LB↔pod hop, plus self-hosted deployments that have no LB at all (Jetty is the edge).

Playwright with h2/h2c — out of scope for this PR: browsers refuse h2c (cleartext HTTP/2) by design. type: h2 (TLS) would work for Playwright but requires keystore + cert-trust provisioning per environment, which is operator infrastructure unrelated to this code change. The default Playwright config still runs against type: http (HTTP/1.1) and is unaffected by anything in this PR. The full HTTP/2-at-the-edge story (browsers → LB) is covered by the runbook in .context/http2-brotli-runbook.md (nginx-ingress / ALB / Traefik annotations).

Companion to PR #28014 (P1: ETag/304, deferred tab fetches, lazy widgets), which is still under review. The two PRs are independent.

Type of change:

Improvement
Bug fix (servlet Vary header + q-value parsing)

Frontend Preview (Loom)

Not applicable for this PR — perceived-latency improvements are best evaluated in a running stack with realistic data.

Checklist:

I have read the CONTRIBUTING document.
My PR title is in the form of <type>: <title> and follows Conventional Commits Specification
My commits follow the Conventional Commits Specification.
I have commented on my code, particularly in hard-to-understand areas.
I have added tests that prove my fix is effective or that my feature works (LineageHydrateIT for the new endpoint, three new OpenMetadataAssetServletTest cases for the servlet fix).
New and existing integration tests pass locally with my changes (mvn verify -pl openmetadata-integration-tests -P postgres-opensearch -Dit.test=SystemResourceIT,LineageHydrateIT 39/39 green; backward-compat verified against type: h2c connector).
UI checkstyle passes locally on touched files.
Java spotless passes locally.
I have updated the API documentation

🤖 Generated with Claude Code

Summary by Gitar

Dependency management:
- Upgraded apache-jena-libs to 5.6.0 and migrated to explicit jena-core, jena-arq, and jena-rdfconnection modules to prevent class-loading conflicts.
- Updated titanium-json-ld to 1.7.0 to ensure compatibility with Jena 5.6.0's RDF JSON-LD API.
Dependency security and maintenance:
- Excluded netty-transport-native-epoll from Azure SDK dependencies to mitigate CVE-2026-42577.
- Bumped azure-kv, redshift-jdbc, simplejavamail, and reactor-netty-http versions.
- Explicitly pinned libthrift to 0.23.0 to address CVE-2026-43869.
- Updated Maven central repository URL to the canonical host.

_{This will update automatically on new commits.}

Replaces N per-node entity GETs with one round-trip. Accepts a list of (type, id) pairs and returns hydrated entities grouped by entityType. Each entity is authorized individually with VIEW_BASIC; entities the caller cannot read are silently dropped (rather than failing the batch). Useful primitive for any client that needs to hydrate a graph of mixed entity types (lineage views, dashboards-of-tables, tag explorers). - Schema: openmetadata-spec/.../api/lineage/hydrateLineageRequest.json - Resource: LineageResource#hydrateLineageEntities - UI helper: hydrateLineageEntities() in lineageAPI.ts - Integration test: LineageHydrateIT Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Tab-switch on /explore (Tables → Dashboards → …) re-runs the same shape of search-fetch with a different searchIndex. Within a session most users flip back and forth without changing the underlying query; this adds a 30s stale-while-revalidate cache keyed by the same dependency string the page already uses to detect "should I refetch?". Cache hit renders synchronously with no spinner, then revalidates silently in the background. - New Zustand store: useExploreCache (capped at 60 entries, 30s TTL) - ExplorePageV1.performFetch wraps the existing setters to capture resolved state and writes to the cache after each fetch resolves Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Replace top-level <Loader /> gates with skeleton placeholders that mirror page chrome — header, tab bar, content card, grid cells. The underlying load time is unchanged, but eye-tracking research is consistent that structured placeholders feel ~30% faster than a centered spinner because users can predict where content will appear. - MyDataPageSkeleton: header band + 4-card grid mirroring landing page - TableDetailsPageSkeleton: breadcrumbs + entity header + tab bar + content card; reusable shape for other entity-detail pages later - LineageSkeleton: row of node-shaped cards so the user perceives "graph is coming" rather than "loading" Cheap implementation: pure antd Skeleton primitives, no new SVGs or theme tokens. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

github-actions · 2026-05-10T16:30:20Z

✅ TypeScript Types Auto-Updated

The generated TypeScript types have been automatically updated based on JSON schema changes in this PR.

Copilot

Pull request overview

Phase 2 of the “perceived latency” work: introduces a new backend primitive to batch-hydrate lineage entities, adds a short-lived SWR-style cache for Explore tab switching, and replaces a few top-level spinners with skeleton placeholders to improve perceived load performance.

Changes:

Added POST /v1/lineage/hydrate (schema + resource implementation + integration tests) to hydrate mixed entity types in one request with per-entity VIEW_BASIC authorization.
Implemented a 30s, 60-entry bounded SWR cache for Explore search results keyed by the existing fetch dependency string.
Replaced full-page loaders with skeleton UIs for Table Details, My Data, and Lineage.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
openmetadata-ui/src/main/resources/ui/src/rest/lineageAPI.ts	Adds UI REST helper for the new lineage hydration endpoint.
openmetadata-ui/src/main/resources/ui/src/pages/TableDetailsPageV1/TableDetailsPageV1.tsx	Swaps initial loader gate to a table details skeleton.
openmetadata-ui/src/main/resources/ui/src/pages/TableDetailsPageV1/TableDetailsPageSkeleton.component.tsx	Adds an above-the-fold skeleton layout for Table Details.
openmetadata-ui/src/main/resources/ui/src/pages/MyDataPage/MyDataPageSkeleton.component.tsx	Adds an above-the-fold skeleton layout for My Data.
openmetadata-ui/src/main/resources/ui/src/pages/MyDataPage/MyDataPage.component.tsx	Uses the My Data skeleton during initial load.
openmetadata-ui/src/main/resources/ui/src/pages/ExplorePage/ExplorePageV1.component.tsx	Adds SWR cache read-through + background revalidation for Explore fetches.
openmetadata-ui/src/main/resources/ui/src/hooks/useExploreCache.ts	Introduces bounded SWR cache store for Explore results.
openmetadata-ui/src/main/resources/ui/src/components/Lineage/LineageSkeleton.component.tsx	Adds a lineage graph canvas skeleton placeholder.
openmetadata-ui/src/main/resources/ui/src/components/Lineage/Lineage.component.tsx	Uses lineage skeleton instead of spinner while loading.
openmetadata-spec/src/main/resources/json/schema/api/lineage/hydrateLineageRequest.json	Adds request schema for lineage hydration batch endpoint.
openmetadata-service/src/main/java/org/openmetadata/service/resources/lineage/LineageResource.java	Implements `POST /v1/lineage/hydrate` with per-entity authorization and type grouping.
openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/LineageHydrateIT.java	Adds integration tests for the new endpoint’s basic behavior.

…ervlet OpenMetadataAssetServlet picks .br / .gz / raw based on the request Accept-Encoding header. Two correctness fixes: 1. Add Vary: Accept-Encoding to compressed responses. Without it a shared cache (CDN, corporate proxy) may serve a brotli body to a client that asked for gzip-only — wrong content for the request. 2. Fix q-value parsing. The previous check did a substring match for "q=0", which incorrectly treated "br;q=0.5" as "brotli disabled" and fell back to gzip / raw. Now parses the q-value as a double per RFC 7231 §5.3.4. Also tightens coding-name match to be exact (so "brand" no longer matches "br") and honours the "*" wildcard. Plus: refresh the stale comment in conf/openmetadata.yaml that said "Response compression disabled" while actually enabling it. Three new unit tests cover the Vary header, the q=0.5 case, and the explicit q=0 disable case. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 6 comments.

Adds the dropwizard-http2 module to openmetadata-service, registering the `type: h2` (TLS) and `type: h2c` (cleartext) connector factories via META-INF/services. Operators can now opt into HTTP/2 termination at Jetty by uncommenting the example blocks in conf/openmetadata.yaml — no SDK changes, no client changes, no runtime flags. Both connector types compose HTTP/1.1 and HTTP/2 protocol handlers on the same port (h2c upgrades from HTTP/1.1; h2 negotiates via TLS ALPN), so adopting them is backward-compatible with existing HTTP/1.1 clients. When this matters: - Self-hosted single-node deploys with no LB in front (Jetty IS the edge); they don't have an LB layer to terminate HTTP/2. - Production deploys that want end-to-end HTTP/2 — multiplexing all the way through instead of LB↔pod fanning out to many HTTP/1.1 connections under high parallelism. - Behind h2c-aware proxies (envoy, nginx with proxy_http_version 2). Verified backward compatibility by temporarily switching the IT bootstrap connector to `type: h2c` and running: - SystemResourceIT → 34/34 pass - LineageHydrateIT → 5/5 pass Total 39 IT tests passed, proving the OkHttp3-based SDK works transparently against an h2c-typed connector via HTTP/1.1 Upgrade. The shipped IT YAML is unchanged (still `type: http`); only operators who explicitly opt in to h2/h2c by editing their own conf/openmetadata.yaml get HTTP/2 at the Jetty layer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

github-actions · 2026-05-10T20:21:40Z

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

gitar-bot · 2026-05-10T20:21:42Z

+    // Case-insensitive containment check — `Vary` values are tokens, not URLs, so a simple
+    // contains is sufficient and avoids the cost of splitting + trimming.
+    if (existing.toLowerCase().contains(value.toLowerCase())) {
+      return;


💡 Edge Case: appendVaryHeader substring match may false-positive on substrings

The contains check on line 280 (existing.toLowerCase().contains(value.toLowerCase())) is a substring match, not a token match. If an upstream filter sets Vary: X-Accept-Encoding-Custom, calling appendVaryHeader(resp, "Accept-Encoding") would incorrectly consider it already present and skip appending. In practice this is unlikely for the standard header names used here (Accept-Encoding), but a token-boundary-aware check would be more correct.

Suggested fix:

// Split on comma, trim each token, and compare case-insensitively: boolean alreadyPresent = Arrays.stream(existing.split(",")) .map(String::trim) .anyMatch(token -> token.equalsIgnoreCase(value)); if (alreadyPresent) { return; }

Apply suggested fix

_{Check the box to apply the fix or reply for a change | Was this helpful? React with 👍 / 👎}

Copilot

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated 3 comments.

+    List<UUID> authorizedIds = filterAuthorizedIds(securityContext, entityType, ids);
+    if (authorizedIds.isEmpty()) {
+      return List.of();
+    }
+    return repo.get(uriInfo, authorizedIds, fields, include);
+  }
+
+  /**
+   * Filter the supplied ids to only those the principal can VIEW_BASIC. Uses the non-throwing
+   * {@link Authorizer#getPermission} so denied entities don't pay for an exception walk —
+   * material when the batch is large (up to 200) and the user has restricted access. The
+   * {@code authorize()} variant throws on deny, which would cost an exception construction per
+   * denied entity; for batches that skew denied, that's measurable overhead and the wrong
+   * idiom (exceptions for flow control).
+   */
+  private List<UUID> filterAuthorizedIds(
+      SecurityContext securityContext, String entityType, List<UUID> ids) {
+    String userName = securityContext.getUserPrincipal().getName();
+    List<UUID> authorized = new ArrayList<>(ids.size());
+    for (UUID id : ids) {
+      ResourceContext<?> resourceContext = new ResourceContext<>(entityType, id, null);
+      ResourcePermission permission =
+          authorizer.getPermission(securityContext, userName, resourceContext);
+      if (isViewBasicAllowed(permission)) {
+        authorized.add(id);


    }
  }, [parsedSearch]);

+  const { getCached, setCached } = useExploreCache();


+  setCached: <T = unknown>(key: string, data: T): void => {
+    const { entries } = get();
+    // Re-set keeps insertion order: drop and re-add so this entry is youngest.
+    entries.delete(key);
+    entries.set(key, { data, timestamp: Date.now() });


github-actions · 2026-05-17T02:02:13Z

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

github-actions · 2026-05-23T16:23:26Z

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

sonarqubecloud · 2026-05-23T17:32:03Z

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

harshach and others added 3 commits May 10, 2026 09:23

Copilot AI review requested due to automatic review settings May 10, 2026 16:24

harshach requested a review from a team as a code owner May 10, 2026 16:24

github-actions Bot added backend safe to test Add this label to run secure Github workflows on PRs labels May 10, 2026

Copilot started reviewing on behalf of harshach May 10, 2026 16:24 View session

gitar-bot Bot reviewed May 10, 2026

View reviewed changes

Comment thread openmetadata-ui/src/main/resources/ui/src/rest/lineageAPI.ts Outdated

gitar-bot Bot reviewed May 10, 2026

View reviewed changes

Comment thread ...tadata-service/src/main/java/org/openmetadata/service/resources/lineage/LineageResource.java

gitar-bot Bot reviewed May 10, 2026

View reviewed changes

Comment thread openmetadata-ui/src/main/resources/ui/src/components/Lineage/LineageSkeleton.component.tsx Outdated

Update generated TypeScript types

524a5ad

Copilot AI reviewed May 10, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings May 10, 2026 16:43

harshach changed the title ~~Perceived-latency P2: lineage hydrate, Explore SWR cache, skeletons~~ Perceived-latency P2 + P3.5 servlet fix: hydrate, SWR cache, skeletons, Vary header May 10, 2026

Copilot started reviewing on behalf of harshach May 10, 2026 16:44 View session

Copilot AI reviewed May 10, 2026

View reviewed changes

harshach had a problem deploying to test May 10, 2026 16:54 — with GitHub Actions Error

harshach changed the title ~~Perceived-latency P2 + P3.5 servlet fix: hydrate, SWR cache, skeletons, Vary header~~ Perceived-latency P2 + P3.5 servlet/HTTP2: hydrate, SWR cache, skeletons, Vary, h2c/h2 opt-in May 10, 2026

gitar-bot Bot reviewed May 10, 2026

View reviewed changes

Comment thread openmetadata-service/pom.xml

harshach temporarily deployed to test May 10, 2026 17:19 — with GitHub Actions Inactive

harshach had a problem deploying to test May 10, 2026 17:19 — with GitHub Actions Failure

Copilot started reviewing on behalf of harshach May 10, 2026 20:19 View session

gitar-bot Bot reviewed May 10, 2026

View reviewed changes

Copilot AI reviewed May 10, 2026

View reviewed changes

harshach had a problem deploying to test May 10, 2026 20:29 — with GitHub Actions Failure

harshach temporarily deployed to test May 10, 2026 20:29 — with GitHub Actions Inactive

Merge branch 'main' into harshach/perceived-latency-p2

ccfa3b6

harshach temporarily deployed to test May 17, 2026 02:10 — with GitHub Actions Inactive

Merge branch 'main' into harshach/perceived-latency-p2

f622665

Copilot AI review requested due to automatic review settings May 23, 2026 16:20

Copilot started reviewing on behalf of harshach May 23, 2026 16:20 View session

Copilot AI reviewed May 23, 2026

View reviewed changes

harshach temporarily deployed to test May 23, 2026 16:31 — with GitHub Actions Inactive

harshach had a problem deploying to test May 23, 2026 16:31 — with GitHub Actions Error

harshach temporarily deployed to test May 23, 2026 16:31 — with GitHub Actions Inactive

Conversation

harshach commented May 10, 2026 • edited by gitar-bot Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes:

Type of change:

Frontend Preview (Loom)

Checklist:

Summary by Gitar

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 10, 2026

✅ TypeScript Types Auto-Updated

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 10, 2026

Uh oh!

gitar-bot Bot May 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actions Bot commented May 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 23, 2026

Uh oh!

sonarqubecloud Bot commented May 23, 2026

Quality Gate passed for 'open-metadata-ingestion'

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

harshach commented May 10, 2026 •

edited by gitar-bot Bot

Loading